Pre-Analysis Plans Have Limited Upside, Especially Where Replications Are Feasible

T he social sciences—including economics—have long called for transparency in research to counter threats to producing robust and replicable results (for example, McAleer, Pagan, and Volker 1985; Roth 1994). Recently, the push for transparency has focused on a few specific policies. In this paper, we discuss the pros and cons of three of the more prominent proposed approaches: pre-analysis plans, hypothesis registries, and replications. While these policies potentially extend to all different empirical and perhaps also theoretical approaches, they have been primarily discussed for experimental research, both in the field including randomized control trials and the laboratory, so we focus on these areas. A pre-analysis plan is a credibly fixed plan of how a researcher will collect and analyze data, which is submitted before a project begins. Pre-analysis plans have been lauded in the popular press (for example, Chambers 2014; Nyhan 2014) and across the social sciences (for example, Humphreys, de la Sierra, and van der Windt 2013; Monogan 2013; Miguel et al. 2014). We will argue for tempering such enthusiasm for pre-analysis plans for three reasons. First, recent empirical literature suggests the behavioral problems that pre-analysis plans attenuate are not a pervasive problem in experimental economics. Second, pre-analysis plans have quite limited value in cases where more than one hypothesis is tested, piloted, or surveyed, and also where null results may not be reported. However, in very costly one-of-a-kind field experiments, including heroic efforts as the Oregon Pre-Analysis Plans Have Limited Upside, Especially Where Replications Are Feasible